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Abstract. This paper considers the problems of the integration of independent manipulator control systems. 
Areas of control of the manipulator are: recognition of objects and obstacles, identification of objects to be grasped, 
determination of reliable positions by the grasping device, planning of movement of the manipulator to certain posi- 
tions with avoidance of obstacles, and recognition of slipping or determination of reliable grasping. This issue is a cur- 
rent problem primarily in industry, general-purpose robots, and experimental robots. This paper considers current pub- 
lications that address these issues. Existing algorithms and approaches have been found in the management of both 
parts of the robot manipulator and solutions that combine several areas, or the integration of several existing approa- 
ches. There is a brief review of current literature and publications on the above algorithms and approaches. The advan- 
tages and disadvantages of the considered methods and approaches are determined. There are solutions that cover 
either some areas or only one of them, which does not meet the requirements of the problem. Using existing approa- 
ches, integration points of existing implementations are identified to get the best results. In the process, a system was 
developed that analyzes the environment, finds obstacles, objects for interaction, poses for grasping, plans the move- 
ment of the manipulator to a specific position, and ensures reliable grasping of the object. The next step was to test the 
system, test the performance, and adjust the parameters for the best results. The resulting system was developed by the 
research team of RT-Lions, Technik University, Reutlingen. The hardware research robot includes an Intel Realsense 
camera, a Sawyer Arm manipulator from Rethink Robotics, and an internally grabbing device. 
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Anotauia. Jlana poOota posriagae mpoOsemu interpalii HesasiexKHHX CHCTeM KepyBaHHA MaHiIlyJIATOpOM. 
OOsacTaMu KepyBaHHA MaHIIyIATOPOM €: pO3Ii3HaHHA 0O’€KTIB Ta HepeliKOL, BU3HaYeHHA OO’EKTIB WIA XalaHHaA, 
BH3Ha4eHHA HaHUX NOSHUii MpHCTpoeM xallaHHA, WwIaHyBaHHA PyXy MaHillyATOpa JO BU3HaYeHUX MOSHI 3 
YHUKHeHHAM Tepelikoy Ta posili3HaHHA IpOKOB3yBaHHA a0o BH3Ha4eHHA HaiMHoro xanaHHa. PosriaHyTe MMTaHHA € 
cy¥acHoro TpoOsemor0, Hep 3a BCe y Tasly3AX MPOMUCIOBOCTI, POOOTIB 3araIbHOrO MpH3HayeHHA Ta CKCIIepHMcH- 
TAJIbHUX poooTiB. Y WaHii pOOoTi posrsAHyTO aKTyaJIbHi MyOMikalli, WO PpO3TATaIOTb 3a3Ha4eHi MMTAHHA. 3HaliyeHo 
HasBHI aIFOpHTMH Ta MAXOAM B yIpaBHHI AK YaCTHHAaMM POOoTa-MaHillyATOpa, Tak 1 PieHHA, WlO OO’eAHYIOTb y 
coOi WeKisbKa oOnacteli, ado inTerpaltii MDK COOOIO AeKIIbKOX ICHYIOUMX MXOWiB. € KOPOTKHM OFIA akTyasbHOI 
JUTepatypH Ta nyOikallii Ha TeMy BULe3raqaHux aITOpHTMIB Ta MiAXoJiB. BusHayeHo MepeBari Ta HEAOIIKH posriisa- 
HYTHX MECTOAIB Ta HiAXoniB. IcHy!oTb PiLIeHHA, WO MOKPHBAaIOTb aOo JeAKi OOacTi, aOo TIIbKH OJHY 3 HX, WO He 3a- 
J{OBOJIbHAC BUMOTaM BHpileHHA TpoOsemu. Kopuctyrouncb IcHYIOYMMH TiAXOAaMu, BU3Ha4eHO TOUKH iHTerpallii ic- 
HYIOUNX peawlizalit Tid OTPHMaHHA Halikpalloro pe3syibtTaty. Y mporeci poboTu Oyo pospoOjeHo cucTemMy, WO aHa- 
JU3Y€ HaBKOJIMMIHE CepeOBHIe, 3HAXOAUTb NepeWiKOH, 06’€KTH JIA B3a€MO/I, MO3H VIA XallaHHA, WIaHye pyX MaHi- 
IlyATOpa JO BH3HAYeHO! 03H Ta BIICBHIOETECA Y HadiMHOMy XallaHHi 0O’exta. HacTylMHHM KpOKOM CTasIO TeCTyBaHHA 
CHCTeMH, MepeBipka Tipal[e3qaTHOCTI NiAXoAy Ta HalaliTyBaHHA MapaMerTpiB Aidt Halikpallux pesyibTaTip. Cuctema, 
CTBOpeHa B pe3yIbTaTi poOoTH, Oya po3ropHyTa Ha AOCIHHUbKOMy poooTi komaHav RT-Lions daxymptety Technik 
Yuigepcntety Poiirminrena. AnapaTHOJOCIHHUbKMM poOoT BKIOYae B Cede Kamepy Intel Realsense, ManitynaTop 
Sawyer Arm Big Rethink Robotics Ta npucTpili xanaHHa BHYTPILUIHbOI pospoOKu. 

Karouosi copa: ROS; xanaHuHa; posi3HaBaHHa 00’€KTIB; YHHKHeCHHA TepeliKON; epeBo OKTaHTIB; XMapa 
TOOK 


Introduction recognition, grasp pose selection, obstacle de- 
Controlling a robot arm is one of the tection, motion planning. 
most important parts of robotic software deve- There are existing researches and deve- 
lopment. It consists of next problems: object lopments for the mentioned problems. But 
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combining all existing solutions and approa- 
ches might be a more complex task. The final 
solution should combine the most appropriate 
solution in each area and bring them together 
as a complete system. 

Problem Statement 

There are several areas in object mani- 
pulation: grasping, motion planning, slip de- 
tection. Communication between them is not 
established. 

Analysis of recent research and 
publications 

The researched problem is relevant in 
robotic arm development. It was found some 
studies and literature mention the stated prob- 
lem. But almost all of them are concentrated 
on independent solutions while the problem 
is the integration of small target solutions in a 
complete system. 

There are several basic approaches to 
control the arm. Including Movelt! [2], point- 
cloud for grasping [11]. 

Authors [10] describe an appropriate 
algorithm for arm self-collision avoidance. 
This paper shows a complete algorithm, but it 
does not show integration with environmental 
collision avoidance. 

Authors [7] describe a complete algo- 
rithmic solution but without implementation. 

The aim of the research 

Develop a method that will find the 
best grasping position and plan a robotic 
arm's c motion from a current position to the 
desired grasping position. After grasping is 
done, a gripper should hold the object with 
enough but not excessive force. 

Main material 

Grasping 

One of the most significant areas of in- 
teraction between robot and environment is 
the manipulation of objects. Manipulation can 
be a part of many highly useful processes: as- 
sembling, sorting, moving, ordering, cargo 
loading etc. These processes are used in in- 
dustrial and casual robots. 

Robotic manipulators should have an end 
effector, also known as End of Arm Tooling 
[4]. Robot end-effectors are subdivided into 
next types: 

e Impactive: physically grasp by applying 
direct force to the object. 
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e Ingressive: penetrate the surface of the 
object to pick it up. 
e Astrictive: attractive forces are applied, 
like vacuum or magnetism. 
e Contigutive: requires direct contact for 
adhesion. 
The considered problem is related to the 
impactive type of gripper. 
Fin-Ray effect 
The Fin-Ray effect is an effect that de- 
scribes a flexible construction that bends 
around an object when force toward the fle- 
xible structure is applied. Such construction 
gives an additional contact area that provides 
additional friction. Rays of fish fins inspired 
the structure. Fish fins have a structure with 
two bones that are attached to each other with 
elastic tissue. The tail fin is the main point to 
apply force for the movement. The tin consists 
of several basic structures stacked one above 
the other. The structure must be light but 
strong enough since excess weight would pro- 
duce unnecessary energy loss. The design, 
which copies a fish fin, consists of two atta- 
ched longitudinal fibers. There are cross fibers 
among the longitudinal fibers that keep the 
whole structure after assembly. The cross and 
longitudinal fibers are connected flexibly, 
which allows the required movement between 
them [5]. 


Fig.1. Fin-Ray effect 


Finger material 

The research team was faced with a 
choice of how to produce fingers for the gripper. 
3D-printing was chosen as the most straight- 
forward way which meets all requirements. The 
fingers are done with FDM 3D printing with 
TPU material. Other parts of the case are made 
with aluminum and ABS plastic. 
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Fig. 2. 3D-model of the used gripper 


Environment perception 

Next, we must detect and store informa- 
tion about the environment. The system should 
scan the environment with perception input de- 
vices, represent it as a data structure and pro- 
cess to avoid obstacles during path planning. 

Since the robotic arm can move in 3D 
space, it should be full of volumetric obstacle 
detection. Usually is done by binocular or mo- 
nocular 3D scanning using SLAM [2; 3]. 

SLAM -— simultaneous localization and 
mapping, is an approach to process multiple 
frames and sensor positions to build a volume- 
ric model of the environment. 

Visual slam can be performed even with 
a single camera (monocular) setup. It is cheap 
and straightforward. Depth is not fully avai- 
lable from one image; instead, we need mul- 
tiple images to match them and compute a 
disparity. 

With two cameras, it is even easier — we 
have a fixed distance between cameras. Ha- 
ving the disparity, we can set a distance from 
the camera for each point and, based on it, 
build a 3D representation of an image for the 
current frame. 

The Intel RealSense depth camera D435 
is a stereo solution, offering quality depth for 
various applications. It is a wide field of view 
that is perfect for applications such as robotics 
or augmented and virtual reality, where the 
most important part is seeing as much as pos- 
sible. The Intel RealSense D435 depth camera 
has a wider FOV at approximately 85° field of 
view. D435 depth camera has a global shutter. 
Cameras with a rolling shutter record all the 
pixels in a scene by rapidly scanning either left 
and right or vertically. This will usually hap- 
pen for a couple of frames, but the data will be 
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saved as a single frame. Global shutter came- 
ras operate differently in that they snapshot the 
whole scene in a single frame, so every pixel 
is captured simultaneously. 

The whole SLAM approach implies 
building a full map. SLAM can use multiple 
different types of sensors, and the powers and 
limits of various types of sensors have been a 
significant driver of new algorithms. Statisti- 
cal independence is required to cope with met- 
ric bias and noise in measurements. Data from 
different types of sensors can be processed by 
different SLAM algorithms whose approaches 
are more compatible with the sensors [6]. 

In our case, we have SLAM with odo- 
metry, lasers, and now, octrees from the point 
cloud. 

Software platform 

To organize the project was used ROS — 
Robot Operating System. It is big framework, 
even similar to the operating system as it cal- 
led, besides it is not an operating system. ROS 
also provides some built-in packages, for navi- 
gaition, simulation, localization etc [1]. 

To test and debug anything without 
working with the real robot we can use Gaze- 
bo which simulates robot in virtual space. 


Fig. 3. Simulation in Gazebo 


Point cloud 

After the robot received any depth data 
from the image, it should store this data. One 
way is a point cloud — a set of points in space. 
When a camera (or it is generated with mul- 
tiple frames) captured a depth image, each pi- 
xel of the depth image can be converted to a 
point in space relative to the camera position. 
Then this data could be added to previously 
generated point clouds from previous frames. 


But if the robot only captures and adds 
the data, data will become bigger and bigger. 
Even when the camera sees the same object, it 
will add more and more points. It is an unne- 
cessary big amount of data. 

Voxels and Octree 

One of the ways to represent a 3D envi- 
ronment discretely is voxels. A voxel is a vo- 
lumetric pixel. Basically, a voxel is a value in 
the regular grid. Usually, the value is a flag if 
it is occupied; additionally, it can contain any 
data, like color. In the typical case, if we have 
a voxel grid, the voxel does not represent a 
real position in 3d space; it contains only its 
position instead. Such an approach can be 
more efficient than more common polygonal 
graphics, but sometimes it can be more com- 
putationally expensive. 

But it also can be optimized — with oct- 
ree. An octree is a tree-based data structure si- 
milar to voxels. It can be understood as va- 
riable size voxels. 

The idea is to make one scene-size oc- 
tant then reclusively divide it into eight octants 
if not all space inside the octant is occupied. 


Fig. 4. Octree 


The OctoMap library implements a 3D 
occupancy grid mapping approach by provi- 
ding data structures and mapping algorithms 
in C++, especially suitable for robotics. In this 
research, I OctoMap as part of Movelt! 
framework. 

Grasp pose detection 

Before grasping any object, the object 
should be recognized and chosen. It is a prob- 
lem in robotics arm control to solve. Approa- 
ches to grasp perception can be subdivided in- 
to the following ways: 
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— With known models. Based on the recogni- 
tion of predefined loaded models. The me- 
thod tries to find similar objects among 
predefined CAD models. It can be more ac- 
curate with known objects but usually does 
not work if the object is not recognized. 

— Without known models. When the system 
allows finding a grasping position for the 
object even if it is an unknown object, but 
at the same time, it can also support the de- 
tection of known objects. 

There is an implementation of the se- 
cond approach — the Grasp Pose Detection 
(GPD) package. It provides 6-DOF grasp po- 
ses for objects. It means that an object can be 
grasped from any side with any orientation if 
the arm can provide such a pose. At the mo- 
ment, it supports only two-finger parallel grip- 
pers, which is appropriate for the gripper we 
use. Besides, the gripper is not a parallel jaw 
gripper, rather a scissors-type [11]. 

As an input, GPD consumes point cloud 
data and outputs available grasp poses. 

The main strengths of GPD are: 

e works for novel objects (no CAD models 
required for detection), 

e works in dense clutter, and 

e outputs 6-DOF grasp poses (enabling more 
than just top-down grasps). 

Arm control with Movelt 

Movelt is an easy-to-use open-source 
robot manipulation platform for commercial 
application development, prototyping, and tes- 
ting algorithms [9]. 

Movelt has a plugin interface to work 
with motion planners. It allows Movelt to use 
and communicate with different motion plan- 
ners which can be provided by multiple libra- 
ries. That makes Movelt easily extensible [8]. 

To represent the actual world around the 
robot and store its and robot’s state the frame- 
work uses a planning scene. It is supported by 
the planning scene monitor inside the move 
group node. The planning scene monitor sub- 
scribes to: 

e State Information: on the joint_states topic; 

e Sensor Information: use the world geo- 
metry monitor described below; 

e World geometry information: gets data 
from user input on the planning scene to- 
pic (as a planning scene diff). 


© B.B. Pyab 


ISSN 2710-1673. Artificial Intelligence, 2020, Ne 3 
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Fig. 5. Planning scene structure 


The Occupancy map monitor is deve- 
loped by using Octomap to represent the occu- 
pancy map of the environment. Octomap can 
encode probabilistic information about indivi- 
dual cells, but this information is not used in 
current Movelt version. The Octomap output 
as octree can directly be passed into Flexible 
Collision Library, the collision checking libra- 
ry that Movelt uses. 

Collision checking in Movelt is configu- 
red inside a Planning Scene with the Colli- 
sionWorld object. Fortunately, Movelt is set 
up so that end users never really have to worry 
about how collision checking is happening in- 
ternally. Collision checking in Movelt is 
mainly processed using the FCL package — 
Movelt's primary collision checking library. 

Movelt supports collision checking for 
different types of objects, including: (1) Me- 
shes; (2) Primitive Shapes — e.g. boxes, cylin- 
ders, cones, spheres, and planes; (3) Octomap 
—octree generated by Octomap can be directly 
used for collision checking. 
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Slip detection 

The easiest way to achieve haptic per- 
ception is to use an acceleration sensor. This 
can be used to record vibrations on the gripper 
in the X, Y and Z directions. The acceleration 
sensor should be mounted as far in front of the 
effector as possible in order to avoid damping 
of the vibration by the elastic TPU. 

In addition, the damping property could 
thereby be used to reduce vibrations from 
outside [7]. 

Another possibility is to record the for- 
ces on the connection flange of the gripper. 
This also enables vibrations on the gripper to 
be recorded in the X, Y and Z directions. For 
this purpose, strain gauges are glued to the 
flange of the gripper. Due to the changes in re- 
sistance, the vibrations that act on the flange 
can be calculated back. In addition, conclu- 
sions can be drawn about the weight of the 
gripped object. The elastic TPU may dampen 
the vibration on the finger so much that it dis- 
appears in the noise. 
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Direct measurement of the gripped ob- 


ject is conceivable with an optical mouse sen- 
sor or a trackball/trackpad fitted in the recess. 
This would make it possible to detect an ob- 
ject slipping out directly. The detour via a 
time-consuming and computationally inten- 
sive evaluation could perhaps be omitted. 


Conclusions 
As a result of the work, the complete so- 


lution was developed. This solution provides a 
full pipeline for detection, motion planning, 
and grasping. 


During the research, existing algorithms 


and solutions were considered. After the full 
analysis of the problem and available solu- 
tions, a complete system that solves the given 
problem was presented. 
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